ERIC ED477039: The Virtual Factory Teaching System (VFTS): Project Review and Results.

DOCUMENT RESUME

ED 477 039

IR 021 767

AUTHOR

TITLE

SPONS AGENCY
PUB DATE
NOTE

CONTRACT ,
AVAILABLE FROM

PUB TYPE
EDRS PRICE
DESCRIPTORS

Kazlauskas, E. J. ; Boyd, E. F. , III; Dessouky, M. M.

The Virtual Factory Teaching System (VFTS): Project Review
and Results.

National Science Foundation, Arlington, VA.

2002-06-00

7p.; In: ED-MEDIA 2002 World Conference on Educational
Multimedia, Hypermedia & Telecommunications. Proceedings
(14th, Denver, Colorado, June 24-29, 2002); see IR 021 687.
EEC-9872488

Association for the Advancement of Computing in Education
(AACE), P.O. Box 3728, Norfolk, VA 23514. Tel: 757-623-7588;
e-mail: info0aace.org; Web site: http://www.aace.org/DL/.
Reports - Evaluative (142) — Speeches/Meeting Papers (150)
EDRS Price MFOl/PCOl Plus Postage.

Cooperative Learning; Distance Education; Higher Education;
Information Networks; ^Instructional Design; Instructional
Innovation; Multimedia Materials; ^Program Evaluation;
Student Projects; ^Teaching Methods; Technical Education;
World Wide Web

ABSTRACT

This paper presents a review of the Virtual Factory Teaching
(VFTS) project, a Web-based, multimedia collaborative learning network. The
system allows students, working alone or in teams, to build factories,
forecast demand for products, plan production, establish release rules for
new work into the factory, and set scheduling rules for workstations.

Included in the paper are a system description and list of project tasks. The
evaluation component involving 3 different campuses is described and the
results of 3-years of analyses are presented, including demographic
descriptions, self-assessment results, performance results, attitudinal
responses, and usability. In addition, instructor observations and course
project output is also examined as components of the VFTS evaluation effort.
It is believed through the analysis that the VFTS is an excellent
instructional method to teach students the integration of the different
modules in operations planning. (Author)

Reproductions supplied by EDRS are the best that can be made
from the original document.

r ^

o^

m

o

Q

W

The Virtual Factory Teaching System (VFTS): Project Review and

Results

PERMISSION TO REPRODUCE AND
DISSEMINATE THIS MATERIAL HAS
BEEN GRANTED BY

Marks

TO THE EDUCATIONAL RESOURCES
INFORMATION CENTER (ERIC)

Kazlauskas, E J.
Instructional Technology
University of Southern California
United States
kazlausk@usc.edu

Boyd III, E ,F.
University of LaVeme
United States
fboydl@fboyd,com

Dessouky, M M.
Industrial Engineering
University of Southern California
United States
magcd@usc.cdu

' U.S. DEPARTMENT OF EDUCATION
Office of Educational Research and Improvement
EDUCATIONAL RESOURCES INFORMATION
CENTER (ERIC)

This document has been reproduced as
received from the person or organization
originating it.

□ Minor changes have been made to
improve reproduction quality.

Points of view or opinions stated in this
document do not necessarily represent
official OERI position or policy.

Abstract: This paper presents a review of the Virtual Factory Teaching (VFTS) project, a
web-based, multi-media collaborative learning network. The system allows students,
working alone or in teams, to build factories, forecast demand for products, plan
production, establish release rules for new work into the factory, and set scheduling rules
for workstations. Included in the paper are a system description and list of project tasks.
The evaluation component involving three different campuses is described and the results
of three-years of analyses are presented, including demographic descriptions, self-
assessment results, performance results, attitudinal responses, and usability. In addition,
instructor observations and course project output is also examined as components of the
VFTS evaluation effort. It is believed through the analysis that the VFTS is an excellent
instructional method to teach students the integration of the different modules in operations
planning.

VO

fN

Background

To address the manufacturing educational needs of new engineers, a web-based, multi-media
collaborative learning network, referred to as a Virtual Factory Teaching System (VFTS), was developed
under an National Science Foundation (NSF) grant (Dessouky, et, al 1998; Dessouky, et. al, 2001;
Kazlauskas, et. al. 2000; Kazlauskas, 2001). This tool is currently being used by engineering students from
the University of Southern California, San Jose State University, the University of Virginia, and most
recently North Carolina Agricultural and Technical State University. The overall aims of the project are:

• To provide, disseminate, and evaluate a manufacturing education pedagogical tool that
promotes student understanding of complex factory dynamics.

• To improve student skills in communication, persuasion, negotiation, and management, as
well as in the technical arenas of production planning, forecasting, simulation, scheduling,
and integration.

• To provide a forum in which engineering and business school students can participate in
virtual teams that cut across universities.

The research plan of the project is aimed at exploring the interface between virtual factories,
engineering education, intelligent agents, and the Internet for new ways of teaching modem manufacturing
problems, practices, theory, and techniques to engineering and business undergraduate students. In

ATtrA THT A TOIT TP

2

addition, it is aimed at examining its potential for use as an information vehicle on the topic of
manufacturing for K-12 students. Various research questions are addressed, such as: how students perform
when using new technology-enhanced modes of learning; what are the effects on attitudes; how intelligent
agents might assume tutoring and participative roles; and how team performance, hampered by
geographical separation, might be enhanced via advanced communication technologies.

The project timetable consisted of the following: a Baseline period to provide an understanding of
the Industrial Engineering courses (with scheduling content) using traditional instructional methodologies,
i,e, without the VFTS; the use of the VFTS to provide an understanding of the Industrial Engineering
courses (with scheduling content) offered at the various institutions; then the use of an expanded version of
the VFTS at the same institutions with pedagogical agents and use of virtual teaming included. The
following presents an overview of the features of the VFTS; a description of the evaluation approaches; and
three-year project results.

System Description

The architecture of the VFTS was kept simple and modular. There are three layers in the design:
AweSim Server, VFTS Java Server and Clients, Clients, which are students in our case, use standard
WWW browsers like Microsoft Internet Explorer, to connect to the VFTS Java Server using its Web Page,
Most of the communication between the clients and the server takes place using Java applets. The Java-
Server functions as a mediator between the AweSim factory servers and the clients. The Awesim Server is
responsible for factory knowledge and simulation. The layers interface using a message protocol set up to
minimize bandwidth requirements. Other components of the VFTS include the use of Ptolemy, a graphing
package, and Drasys, an Operations Research (OR) package.

The system allows students, working alone or in teams, to build factories, forecast demand for
products, plan production, establish release rules for new work into the factory, and set scheduling rules for
workstations. They can run simulations where an animated panel displays jobs progressing through their
factory, with queue counts, finished goods counts, graphs, and reporting functions all available. Students
access via the VFTS, using computers in university computer labs, or in their dorms or homes, a virtual
representation of the factory. The professor posts assignments related to this factory over the Internet,
“unfreezing” parameters as necessary so students may experiment without redefining the entire factory.
Students observe the effects of their decisions, and student teams assume factory roles to solve problems; if
they reach an impasse, intelligent agents provide guidance. Selective information may be given based on
student roles; for example, the production supervisor may have equipment information, the engineer new
technology information, etc. Students sort out strategies and can discuss options via e-mail and electronic
chat rooms. Since this course is a common one found in many universities, collaboration among
universities is feasible. Faculty members virtually “team teach” the course. Intelligent agents are
incorporated into the VFTS to monitor student progress and provide immediate feedback.

The latest version of the VFTS software includes support for user account management, adding
security to allow students within a group to share data while preventing access to it from students outside
the group, a more easy to use interface, instrumentation so that the software will gather data on student
usage patterns, on-line help, on-line documentation and an on-line tutorial to help for students and faculty
learn how to use the VFTS, and an introductory homework exercise to help students learn how to integrate
the VFTS into the course, A project that uses the VFTS was developed to complement the students'
classroom learning and was integrated into the courses at the participating universities. The latest version
also includes a pedagogical agent that monitors students' use of the VFTS and provide guidance. This
required integrating the pedagogical agent software (ALI) into the VFTS, adding the ability for ALI to
maintain a persistent model of each student's knowledge across sessions, extending AU to include more
sophisticated explanation capabilities, adding VFTS-specific knowledge to allow ALI to understand the
instructional objectives of the VFTS and provide appropriate support to students, and instrumenting ALI to
maintain a log of interactions with students to aid our evaluation.

The current version VFTS is available at http://vfts,isi,cdu .

3

Project Tasks

The VFTS project included a large set of tasks involving design, development, usage, evaluation,
and dissemination. The various tasks associated with the VFTS project are as follows:

■ Define instructional objectives and complete evaluation design.

■ Develop and/or acquire evaluation instruments.

■ Solicit feedback on the instructional objectives, evaluation design, and VFTS use from each
university.

■ Gather evaluation data for control group (engineering classes without the VFTS).

■ Analyze evaluation results for control groups.

■ Complete instrumentation of the software to support evaluation

■ Teach engineering classes with the VFTS and gather evaluation data.

■ Analyze evaluation results for experimental group and compare to control group.

■ Use the evaluation results to revise and refine the VFTS and its use in the engineering
curricula, its instrumentation, and the pedagogical agents.

■ Teach engineering classes with the VFTS, including the use of virtual teams that span
multiple universities, and gather evaluation data for this final experimental group.

■ Analyze summative evaluation results.

■ Make final revisions to the VFTS software based on evaluation results.

Evaluation

In the preceding list of project tasks, it should be noted that considerable emphasis is placed on the
design and implementation of an extensive, multi-university evaluation, which is a central component of
the VFTS project. The initial evaluation efforts were used a) to check to make sure each project step was
implemented according to plan and that milestones are being met; and b) to consider project modification.
This evaluation, formative in nature, assisted in modifying the design and development of the evaluation, as
well as of the overall design of the web-based VFTS. For example, a Computer Competencies Multiple
Domains instrument that was used initially to determine the level of entering competency in such areas as
spreadsheets and statistical packages was dropped. It was determine through initial surveys that the learners
were homogeneous in terms of a high level of computer competencies. Through the evaluation process, it
was decided to gather more data on such issues on the ease of use and operation of the web-based VFTS,
and to modify the actual design by changing the opening screen, and by including more tutorial help.

An emphasis of this project is to evaluate learner outcomes, attitudes, and learning. This effort
includes the use of various instruments and examination of student work.

The following are the instruments that are presently being used in evaluation:

• Student Pre -course and Post-course Survey instruments which are used to gather data on
student demographics and student self-rated assessment of entering/post-course knowledge of
operations scheduling, such as the skills, knowledge, and abilities in the use of gantt charts
and regression and time series models .

• Operations Scheduling Pre-test/Post-test instruments which are used measure course content
before and after instruction and use of the VFTS.

• Affective-Level/Attitudes Instrument (VFTS Participant Opinion Survey) Pre-test/Post-test
instrument which are used to measure reactions to the use of the VFTS, such as reactions to
use of simulations, working collaboratively, and relevance to future career.

• Usability Evaluation instrument which is used to gather data on such issues on the ease of use
and operation of the web-based VFTS.

The VFTS Studies

Various studies were conducted in the VFTS Project. The first of these was the Baseline
evaluation conducted at the various universities, to provide an understanding of the Industrial Engineering

courses (with scheduling content) using traditional instructional methodologies. The second of these was an
investigation of the use of the VFTS at the same universities, to provide an understanding of the Industrial
Engineering courses (with scheduling content). A second usage of the VFTS occurred, an enhanced version
with teaming and pedagogical agency. The following represents the findings from these investigations (at
three universities to-date).

Demographics

For the most part the demographic analyses showed similarities between both the Baseline and
VFTS groups at the various universities, and the demographics appear to parallel those that are found in
most engineering programs. Most students were in the 21-22 age category, with the age distributions at
most universities typical of a ‘traditional’ undergraduate student. However, students at one institution had
an older mean and the age range was quite large. At all institutions, students were mostly seniors. The
majority of students at all three campuses were male, as reflects the common engineering enrollment
pattern. There was no significant difference on the gender variable between the groups. The primary
language was English but many other languages were represented, with 50% or more students at two
institutions having a language other than English as their first language.

Self-Assessment

Instruments gathered pre and post data on a student’s interest in the course and on their self-
assessment of knowledge of course content, such as whether they could apply forecasting methods to new
problems.

Students have a moderate interest in the course. One can support the general statement that
students know little, if anything, about the content of the course but think that it will help them in a more
general way. At two institutions, students indicated that they did see the course assisting in developing
industrial engineering fundamentals. Pre and post-test measures of self-assessment indicate significant
gains in most areas. In particular, the overall gains in self-assessment in the second year offerings of the
VFTS at two universities were significant. The self-report assessment provides evidence that the students
feel as if they have learned the subject in the course.

Performance

As a general statement, it seemed desirable to determine if there are significant differences
between and among the participant groups. Representative analyses reflect: performance at each institution
in both the Baseline and VFTS options; and Baseline to VFTS Comparisons. In examining Baseline and
VFTS groups at the various institutions, analysis indicated overall gains in learning in each offering of the
course between pre-test and post-test, and in specific content areas of scheduling. There appears to be a
significant difference among the schools on the pretest measure, with one institution scoring significantly
higher than the others. Comparison between the Baseline and VFTS performance at each institution
indicated a difference on selected pre-test items. Correlations were calculated for the pretest instruments.
Baseline to VFTS groups. Although there were some correlations between age and certain items and among
certain items, it appears that these results were of no significance in this particular study. There were no
significant difference between the Baseline and the VFTS groups on the final exam except for a few
selected items. As noted, the VFTS groups did not perform as well on the pretest as the Baseline groups,
yet scored approximately the same on the final examination. This could imply that the less strong content-
knowledge groups, using the VFTS, performed equivalent to the stronger content-knowledge groups. The
post-test scores at one institution are significantly higher than at the other universities. We remark that at
only this university was the post-test a component of the student’s final grade. In fact, it was treated as the
final examination so in this case the students took it much more seriously where they most likely studied

er|c

5

for the examination. A comparison of the post-test scores between the baseline and VFTS groups at this
university show a significant gain for the VFTS group, although for a smaller class size.

Attitudes

Instruments gathered pre and post data on a student's opinions, on such scales as usefulness,
interest, importance, difficulty, and confidence, for example whether students thought learning to use
computer simulations to work on class assignments would be useful to their future career. Results of the
analysis indicated that there was no significant change in the attitudes of the students, even with the use of
the VFTS.

Usability

In regard to the use of the VFTS, most students seem to be using Internet Explorer on a Windows
98 platform. Student comments on the usability of the software isolated several problems that were
resolved in the updated version of the VFTS. The various modules well- received; for the most part the
online documentation and tutorials not used or students commented that they were of limited value;
students were not positive towards the use of the VFTS teaming aspect of the course. However, effort is
being made to modify the teaming aspect of the course with the assertion that this may impact usability
findings on this topic.

Other Results

Input was solicited from the instructors of the courses at the various universities. Their comments
regarding the use of the VFTS were positive. They noted, in particular, that students using the VFTS had a
more realistic course project than their counterparts in the Baseline classes.

The NSF funding for this project has supported five graduate students at the University of
Southern California. These students have gained valuable experience in Web-based software development,
user interface principles, pedagogical agent technology, educational materials development, and
educational evaluation. Perhaps the most valuable training for all the students is the close multidisciplinary
collaboration between computer scientists, engineering educators, and educational researchers and
educators. These students will have a much broader perspective on research than typical graduate students
that work closely with others only in their own field.

Conclusion

The VFTS was instrumental in facilitating students’ understanding of the integration between the
models for forecasting, production planning, material planning, inventory planning, and scheduling.

It was demonstrated through an analysis of project reports that students had a better understanding
of the integration between the different modules. This was illustrated by the fact that students clearly spent
a good part of their time, with the VFTS, in testing many different scenarios, and students were able to see
the impact of changing a parameter on the output. It is suggested that another major benefit of the VFTS is
that a more complex system can be modeled than otherwise. Thus, it was seen that students had a more
realistic project than their counterparts in the Baseline groups. To summarize, it is believed through the
analysis that the VFTS is an excellent instructional method to teach students the integration of the different
modules in operations planning.

o

ERIC

6

References

Desouky, M. Bailey, D,, Verma, S, Adiga, S. Beckey, G and Kazlauskas, E. (1998) A virtual factory teaching system in
support of manufacturing education. Journal of Engineering Education. 459-467.

Dessouky, M., Verma, S., Bailey, D.E., Rickel, J. (2001). Methodology for PevelopinH a Web-based Factory
Simulator for Manufacturing Education . HE Transactions, 33(3), 167-180.

Kazlauskas, E.J., Desoukky, M., Rickel, J & Johnson, L, (2000) A Web-based Factory Teaching System: Design and
Development. Proceedings of WebNetWOO: World Conference of the WWW and Internet. San Antonio TX, October
30- November 4, 2000,

Kazlauskas, E,J, (2001). Integrating University-based Technology Projects into Education Programs, In Lee, C, (Ed,)
Enhancement of Quality Learning through Information and Communication Technology. The 9th Annual International
Conference on Computers in Education. Seoul, Korea: ICCE, 269-275.

Acknowledgement

This research is funded by the National Science Foundation EEC-9872488.

o

ERIC

7

BEST COPY AVAILABLE

U.S. Department of Education
Office of Educational Research and Improvement (OERI)
National Library of Education (NLE)
Educational Resources Information Center (ERIC)

NOTICE

Reproduction Basis

This document is covered by a signed "Reproduction Release (Blanket)"
form (on file within the ERIC system), encompassing all or classes of
documents from its source organization and, therefore, does not require a
"Specific Document" Release form.

This document is Federally-funded, or carries its own permission to
reproduce, or is otherwise in the pubhc domain and, therefore, may be
reproduced by ERIC without a signed Reproduction Release form (either
"Specific Document" or "Blanket").

o

ERIC

EFF-089 (1/2003)